fix(subscriber): propagate handler error to engage startSubscriber backoff#3218
Open
NitinKumar004 wants to merge 2 commits into
Open
fix(subscriber): propagate handler error to engage startSubscriber backoff#3218NitinKumar004 wants to merge 2 commits into
NitinKumar004 wants to merge 2 commits into
Conversation
NitinKumar004
force-pushed
the
fix/subscriber-error-swallowed
branch
from
9f5b862 to
6a6435d
Compare
Umang01-hash
requested changes
Apr 3, 2026
Member
Umang01-hash
left a comment
Umang01-hash
left a comment
There was a problem hiding this comment.
- Previously:
{"level":"DEBUG","message":{"messageValue":"{\"orderId\":\"ORD-001\",...}","topic":"order-logs",...}}
{"level":"INFO", "message":"handler received order #1: {OrderId:ORD-001 Status:FAILED}"}
{"level":"ERROR","message":"error in handler for topic order-logs: handler: downstream DB unavailable"}
↑ ONE error log, then immediately next message:
{"level":"DEBUG","message":{"messageValue":"{\"orderId\":\"ORD-BEFORE-FIX\",...}","topic":"order-logs",...}}
{"level":"INFO", "message":"handler received order #2: {...}"}
{"level":"ERROR","message":"error in handler for topic order-logs: handler: downstream DB unavailable"}
↑ NO DELAY — tight loop!
After fix (return err — PR change):
{"level":"DEBUG","message":{"messageValue":"{\"orderId\":\"ORD-001\",...}","topic":"order-logs",...}}
{"level":"INFO", "message":"handler received order #1: {OrderId:ORD-001 Status:FAILED}"}
{"level":"ERROR","message":"error in handler for topic order-logs: handler: downstream DB unavailable"}
{"level":"ERROR","message":"error in subscription for topic order-logs: handler: downstream DB unavailable"}
↑ TWO error logs for the same error — from handleSubscription AND startSubscriber
↑ Then 2-second delay before next message is attempted
{"level":"DEBUG","message":{"messageValue":"{\"orderId\":\"ORD-BEFORE-FIX\",...}","topic":"order-logs",...}}
↑ Next message 2 seconds later ✅
Every handler error fires both "error in handler for topic..." and "error in subscription for topic...". In a busy subscription loop under real failure conditions this doubles alert noise.
Maybe we can try a small fix: remove the Errorf on subscriber.go:70 and let startSubscriber's existing log (line 38) be the single error entry point.
That keeps error propagation clean and logs the error exactly once.
NitinKumar004
force-pushed
the
fix/subscriber-error-swallowed
branch
from
fa8abeb to
4e980a6
Compare
Member
|
Conflicts with #3424 (merged) — needs rebase. After rebase the Minor:
|
…ckoff Previously, when a handler returned a non-panic error, handleSubscription logged it and returned nil. That kept startSubscriber's loop tight: it never tripped the err != nil branch, so the 2s backoff never engaged and the loop hot-spun. This change keeps the existing error log and returns the error so startSubscriber backs off for 2s before re-subscribing. The recovered- panic sentinel (errSubscriberHandlerPanic, added in gofr-dev#3424) still returns nil so panicRecovery's log isn't duplicated. Also reset the loop's delay to 0 after a successful iteration — without that, a single failure pinned subsequent successful iterations at the 2s cadence forever. Note: this does not change commit semantics. The handler-error path never reached msg.Commit() before; the broker still redelivers.
NitinKumar004
force-pushed
the
fix/subscriber-error-swallowed
branch
from
702b6c1 to
567d369
Compare
NitinKumar004
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
handleSubscriptionlogged handler errors withErrorfand returnednil. Because the error never reachedstartSubscriber, itserr != nilbranch — which setsdelay = 2 * time.Second— never engaged, so a failing handler caused a tight resubscribe loop on every consecutive failure.This PR:
handleSubscriptionsostartSubscriber's existing 2s backoff engages on failure. The recovered-panic sentinel (errSubscriberHandlerPanic, added in fix(subscriber): emit error on handler panic recover to prevent commit of not processed messages #3424) still returnsnilsopanicRecovery's log isn't duplicated.delayto0after a successful iteration instartSubscriber. Previously a single failure pinned all subsequent successful iterations at the 2s cadence.Rebased on top of #3424.
Fixes #3215
Test plan
TestSubscriptionManager_handleSubscription_HandlerErrorDoesNotCommit(from fix(subscriber): emit error on handler panic recover to prevent commit of not processed messages #3424) is updated to assert the handler error propagates viaerrors.Iswhile still asserting no commit happens.SuccessCommitsandPanicDoesNotCommitfrom fix(subscriber): emit error on handler panic recover to prevent commit of not processed messages #3424 remain.